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PREFACE 


Hie Examination Research Cell (ERC) of the Association of Indian Universities 
has been from time to time investigating into various fundamental aspects of university 
examinations Internal Assessment* Grading* Test and item, analysis, Practical Exami¬ 
nations to name only a few. Results of these research projects have been reported in 
the form of Monographs some of which have been revised to include experiences of 
teachers/colleges/universities during implementation, in subsequent editions. 


At the same time, certain research studies have been conducted and it is felt 
that these have to be reported in the form of research Abstracts and three such ab¬ 
stracts are now getting ready. 

In this third Research Abstract, the following in depth studies have been 
included. 

1 Computer Generation of Equivalent Multiple Choice Items 

2. Computer in Storing and Retrieving Question Banks 

3. Random Matrices Generation 

4. Computer Programme for Analysis of objective Type Test 


It is hoped that teachers, papersetters, examiners and other will find the 
research studies and the results and conclusions will be very helpful and useful and 
that they will be guided to better evaluation of their students’ performance. 

Constructive suggestions for advancing some of these studies, will be most 
welcome. 


New Delhi. 

22nd July, 1978. 


V. Natarajan 
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COMPUTER GENERATION OF EQUIVALENT MULTIPLE CHOICE ITEMS 


The application of the computer to the preparation of student evaluations poses 
some of the same problems as other applications of computers. To obtain flexibility 
one can store enormous volumes of data in fixed formats for rapid retrieval or one 
can develop a few complex algorithms to generate a large quantity of relatively dif¬ 
ferent quest ions/items. In one case* the cost is in storage and in the other the cost 
is in the programme logic and computation. One can develop an application that uti¬ 
lizes features of both of these approaches. 

There are three parts of a multiple choice item viz the stem, the correct 
answer (or key) and the incorrect answers (foils or distractors). It is possible now 
to design an item Pool Management System that permits an association of upto 7 
correct answers and seven distractors with each question stem. Below is given a 
sample of a question with 7 viable answers and seven distractors. 

SAMPLE 


Which one of the following expressions or phrase must represent a rational 
number? 

A 1 a repeating decimal 

A 2 a terminating decimal 

A3 3.1415 

A 4 0.333333 

A 5 12/25 

A 6 (. 101001000100001.. ..) + (3. 010110111011110.) 

A 7 the square root of 4/9 

D 1 a non terminating, non repeating decimal 

D 2 the square root of 3 

D 3 the square root of -4 

D 4 0.232332333233332. 

D 5 35/(7 + (0. 7)) 

D 6 25 / /aT 

D 7 Pi 

If one assumes only one of the multiple choices will be correct, there are 245 
ways that a choice can be made of one of the seven correct answers, followed by four 
of the distractors. Since IPMS also randomizes the arrangement of these five choices, 
there are a total of 29,400 various physical formats that can be generated from this 
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single question. 

It would be unwise to believe that one question could serve as 29400 distin¬ 
guishable evaluation items but experience In classroom applications has indicated 
this to be a practical approach in making the compromise between storage and 
total item generation. With sufficiently large pools or banks of quality items, they 
can be used for pretests, self tests, practice sets as well as post evaluations. 

Even if a student should gain access to the question bank he could not depend on 
memorizing a list of letter answers. Indeed he would have to become involved 
in the content of each question. If the pool or bank is sufficiently large and well 
designed, such access may even serve valid educational objectives: 

Some possible areas of subject matter that lend itself to this kind of treat¬ 
ment of computer generation of equivalent items are: 

The stem may list a law / theory / or a principle / or a concept. 7 applica¬ 
tions are given; 7 things which are not applications but can be mistaken to be 
applications can also be given. This will again generate 29,400 multiple choice 
items. We may think of subject area in perspective; 7 characteristics may be 
identified and given; 7 non - characteristics can also be given. The stem may 
direct the student to either identify a characteristic/ or a non characteristic. 

The actual programme listing is given below: 
to enable the teacher/reader to make use of it in his own area. 

General Methods used in Item-Generation Programme, 

This item generation programme is quite general and can be used for any 
number of rational, irrational choices and the programme takes the vector of Rational 
and Irrational choices together with the number of options required to take in a 
generated multiple - choice question on the input parameters. 

At the first step it calculates the value of P ! where P is the number of op¬ 
tions required to have in the questions. The feature of this value lies in the fact that 
of we have only one choice of a question containing P options, we can generate P ! 
questions, just changing the position of this entities. Then the programme produces a 
Matrix MGR of order (PxP !) with the help of two permutation IP1 on IP2 which 
contains all the P ! different permutations of P numbers. This is accomplished by 
calling SSP subroutine ’PPRCN’ for generating permutations. 

Further it generates all the elements of 2 N Factorial Experiment in the 
standard form where N is the length of vector of irrational choices just as for 2 J 
Factorial Experiment it will generate 8 elements as 

000 100 

001 101 

010 110 

Oil 111 

and of this, the matrix B is formed. 
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N 

Then out of these 2 Factorial Experiment, elements (with entities 0 or 1). The 
matrix FM is formed as containing only these rows of B matrix, whose row sum 
of elements came out as P-1 i. e. one less than the number of options required, 
in the question and this one is excluded and is kept for inserting the Rational - 
Choice, 

As a generated question requires to have P-1 irrational choices and this is 
done taking the help of the generated matrix FM with the rule that 


if FM (I,J) = 1 

; Ith component of vector of irrational choice should 
be included in the jth generated question. 

if FM (I, J) = 0 

; It is not taken 


Proceeding like this we can have x M^P - 1) total questions; where M 
is the length of the vector of irrational cnbices. 

Finally w. r. t. each generated question (P) ! questions can be generated just 
changing their relative positions taking the help of the Matrix MGR. Thus for N 
rational choices, M: irrational choice and P: the option required we are able to 
have. 


(N ) (M ) xP ! 

1 P-1 

total generated Multiple-choice items. 


ITEM GENERATION PROGRAMME 

Dimension R(4), RR(8), MGR (24,4), B(256,8), FM(256,8) 

VE(4) RA(24,4), IP1(4), IP2 (4), IP3 (4). 

READ 22, N, M, NO 

22 FORMAT ( ) 

AN = N 
AM = M 
ANO = NO 

READ 1, (R(I) , I = 1, N) 

FORMAT ( ) 

READ 1, ( RR(I)» I = 1, M) 

L = NO - 2 
DO 40 I = 1, L 
IP1 (I) = I + 1 
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40 IP2 (I) = I + 2 

IPl((No-l)) = NO 
IP1 (No ) = 1 
IP2 ( No ) =1 
IP2 (( L + 1 )) ? 2 
DO 30 J = 1, No 
MGR (1. J) = J 
MGR (2, J) =IP1 (J) 

30 MGR (3, J) = IP2 (J) 

DO 201 =4, N OF 

CALLPPRCN (IP1, IP2, IP3, NO, 2, IER) 
DO20J = 1, NO 
MGR (I, J) = IP3 (J) 

20 IP2 (J) = IP3(J) 


C CALCULATING FACTORIAL (NO) 

NOF =1 
DO 111 I = ], No 
111 NOF = NOF * I 


C 


12 


13 

11 


GENERATING FAC. EXP. ELEMENTS 
DO 10 J = 1, M 
IN = Z ** (J-l) 

K = 2 ** (M - J + 1) 

JP = K - 1 

DO 11 L = 1, JP, 2 
K = (L - 1) IN + 1 
IK = L IN 
DO 12 I = K, IK 
B (I, J ) = 0. 

K = IK + 1 
IK = (L + 1) * IN 
DO 13 I = K, IK 
B (I, J) = 1. 

CONTINUE 
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10 CONTINUE 

C SORTING THOSE ROWS OF B HAVING ROW SUM (NO - 1) 

LA = 2 * * M 
AP = 0. 

DO 100 I = I, LA 
SI = 0.' 

DO 101 J = 1, M 

101 SI = SI + B (I, J) 

AS = NO- 1 

IF (SI. NE. AS) Go To 100 
AP = AP + 1. 

DO 102 J = 1, M 

102 FM (AP, J) = B (I, J) 

100 CONTINUE 

PRINT 500 

500 FORMAT (1H2) 

PRINT 501, (R (I), I = 1, N), (RR(I), I = 1, M) 

501 FORMAT (//// 5x, ’RATIONAL CHOICES’, 4F 4. 0, 

/// ox, ’IRRATIONAL CHOICES’, 8F 4. 0) 

IAP = AP 
Do 201 I = 1, N 
VE (1) = R (I) 

Do 201 J = 1, IAP 
IK = 1 

Do 202 L = 1, M 

IF ( FM (J, L) . EQ. O. ) Gc To 202 
IK = IK + 1 
VE (IK) = RR (L) 

202 CONTINUE 

Do 2010 U = 1, NOF 
Do 2010 I JR = 1, NO 

2010 RA(IJ, I J R ) = VE (MGR ( IJ, IJR)) 

PRINT 203, ((RA (IZ, JK), JK = 1, NO), IZ = 1, NoF) 
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203 FORMAT (/// 5x, (4F 6. 0)) 

201 CONTINUE 

END 


N 

M 

NO 

R 

RR 

NOF 

B 

FM 


VE 


RA 


IP1, IP2, 
IP3 

MGR 


Description of Variables Used 


Number of RATIONAL - CHOICES 
Number of IRRATIONAL - CHOICES 


Options required in a generated multiple-choice item. 

Vection of length N, of Rational-Choices 
Vection of length M, of irrational choices 
Value of Factorial (NO) 

Matric of order (2 x M) of the generated Factorial Experiment 
elements. 

Matrix of order (L x M) where IX2 M , is the matrix derived, from 
B consisting of all those rows of B having row sum as (NO-1) 

Vectors of length No, is a generated item, like this N x M 
(NO-1) vectors are generated. c 

Matrix of order (NOF x NO), one such matrix is associated with 
each generated VE consisting of all generated questions that can 
be generated from the associated VE w. r. t the position of entitles. 


Each is a Vector of length NO ; Dummy vectors required for gene¬ 
rating permutations. 

Matrix of order (NOF X NO) consisting Of all the permutation of 
given NO elements. 


Item generation can be attempted with less number of keys, less number of 
distractors also. Table below shows the range: 


S. No. 

Number of keys/ 
Number of dis- 

3 option 
item 

4 option 
item 

5< f lon REMARKS 
item 


tractors 




1 

4 

4 

144 

384M 

480 

2 

5 

5 

300 

1200 

3000 

3 

6 

6 

540 

2880 

10800 

4 

7 

7 

889 

5880 

29400 
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Assuming all of the other factors related to using a computer for test generation 
are present> one obstacle could still prevent any widespread application. If this system 
requires any significant training problems, its value is diminished. To that end IPMS is 
Itself tutorial In data entry and materials production one can use the system in the ab¬ 
sence of any technical knowledge about computers and it does not require a trained 
person to stand close by for assistance. Everything that a teacher needs to know to 
produce materials from existing question pools can be contained lucidly and legibly on 
one side of 8.1/2" x 10" card. It is possible to design the system without professional 
programmers and using parent volunteers, students and para-professionals to enter 
and edit data supplied by teachers. 

Much testing goes on in education but evaluation as an end in itself is not justi¬ 
fied. IPMS encourages the user to classify each item in terms of 3 specific attributes. 
Choices such as chapter/section/objective or unit /subunit /objective are common 
choices for these attributes. 

In the process of implementation, one may face the problem of changing teacher 
attitudes as they relate to the computer and individualization of instruction. Any inno¬ 
vation in Education must allow a teacher to begin at his own level of professional 
skill and grow into application as his skill Improves. Although a result may be revo¬ 
lutionary, its implementation must be evolutionary. Secondly the importance of 
quality must be understood. 

Finally no matter how great the system may seem to the author, and no 
matter how enthusiastically they are able to describe it, its ultimate success will 
be determined on the basis of whether the apparent value received is greater than 
the perceived effort of using it. Technology in education must be a serving tool and 
not an end in itself. 
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COMPUTER IN STORING AND RETRIEVING 
QUESTION BANKS 


Assistance in preparation of tests and exercises is the emerging use of computers 
that has had very little publicity but which shows signs of widespread appeal. Computer 
assisted test costruciion using the retrieval process is a service now available in some 
developing countries. Computer assistance may involve any combination of several acti¬ 
vities. 

a) The questions may be banked by a computer (and subsequently retrieved) 
individually to store questions, tests and exercises. 

b) Questions may be classified according to content ami statistical characteristics 
so that tests/examinations meeting carefully defined specification can be coni- 
figured automatically and in some cases. 

c) The items/questions may be generated by the computer. 

As one tries to generalise by reviewing the existing systems he is likely to be 
struck first by their diversity. Systems having similar test preparation functions often 
were developed for different reasons. Systems which were motivated by similar consi¬ 
derations often support somewhat different functions. Some systems restrict themselves 
to support test preparation, others offer test preparation as one of the several services 
such as test scoring. Most systems are designed to support evaluation of diagnosis, some 
also provide tests of question for drill or review or for self evaluation. Most computers 
assisted test preparation takes place in post secondary level institutions. Some depl with 
postgraduate certification of the military services. USA leads in exploring this field, there 
are similar activities either underway in Britain, Canada, Germany and Australia. 

Such a service is now available to all the faculty-members and students of the 
California State University and colleges. The Soerates system is available to all 16, 000 
faculty members and more than 290, 000 students, over their 19 campus computer net¬ 
work spanning a distance of 12, 000 km. There are eleven banks from which to draw test 
items. The size of each bank varies from a few hundred, as in case of the ’Indtoductory 
Psychology Bank* to over 15, 000 items as in the case of ’American History Bank’. Tests 
and other exercises can be generalised interactively by students and faculty at time 
sharing terminating through their interactive computer network. The system is function¬ 
ing smoothly enough now so that, the test requested by telephone before 8 P. M., typi¬ 
cally are delivered by the following morning. 

In principle, the generation of tests by selecting questions from large collections 
is not new. It is tradition among teachers to keep collection of items in file-boxes or other 
containers and periodically to deliver stacks selected from these collections to a secre¬ 
tary to be typed. Moreover, computerised item-banks have been used by National Testing 
Services for about two decades. However, the availability of item-banks through computer 
network is beginning to emerge as a novel and powerful tool of the individual classroom 
teacher. 
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In India* some very recent attempts have been made by AIU (Research Cell) for 
the development of 'Question Bank* - the first step of this process. The need of this was 
felt when educationists have come round to the view that much of the fault lies in the 
manner in which the question-papers are set. They have quoted with approval the report 
of the 1966 Education Commission which stated ’’Most of the weaknesses in the present 
system of examinations are due to defects In the questions and questior-papers. Very few 
of the paper setters possess the necessary knowledge and skill in construction of valid and 
reliable tests’. 

The Commission urged that question-papers should be designed to test not merely 
the acquisition and knowledge by the candidate* but also development of his problem-sol¬ 
ving abilities. There are numerous mistakes in grouping of questions and in framing the 
questions, besides there are factual and contextual errors. Often the questions set for an 
examination are out of course, or not quite comprehensible to the average student. Then 
comes the trouble walkouts, tearing of answer books and other hostile demonstrations, 
many of which result in smashing of chairs and tables, injuries to centre superintendents 
and much loss in other ways. In order to f>nd this national waste and remove the general 
complaints of teachers and students, the University Grants Commission has suggested the 
Question Bank scheme. A 

As a result, AIU (Research Cell) has generated fairly big volumes of questions of 
different kinds. This work is meant to be an aid to teaching. Questions have been pre¬ 
pared by hundreds of university teachers in workshops, organised in different parts of 
the country. Categories like multiple-choice, matching, rearrangement, short answer, 
long answer etc. The questions aim at testing fifteen different objectives, like knowledge 
of specifics, trends, comprehension, application, analysis etc. A checklist of the cri¬ 
teria used for prevalidation of question and another to judge the quality of supplementary 
type text-items are also printed. 

The key feature of each bank is the subject matter content classification system. 
Although experience has shown that classification system will reflect at least to some 
degree the personal viewpoint of item-bank co-ordinators, therefore all of the bank co¬ 
ordinators have been encouraged not to incoporate any material that could be specifically 
associated with particular pedagogical philosophy. 


Item Selection 


In most test construction systems questions are stored in computer manipu- 
lable form, so that the test itself can be easily printed. The usefulness of such systems 
depends of course upon the existance of data banks of machine readable questions. 

Many systems are designed only to store and print questions that are speci¬ 
fically requested by the user. However,many other include computer-assistance in 
the selection of items to appear on a test. In these systems items are some how 
classified or associated with attributes. Then attributes may involve either charac¬ 
teristic assigned, like indices of discrimination. 
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To get assistance in selecting items, the user typically requests those having 
the characteristics, he desires. If more are available than were requested according 
to the set norms, the system generally selects at random from those eligible. 

Consequently, when the system selects questions by attribute, the user does 
not usually have control over which the specific item will be retrieved. In such oases 
it is often necessary to revise the test produced initially. Test modification can be 
facilitated by arranging for the computer to remember the test and on the user to 
instruct it to delete items or to add items* 

A few systems aid in test preparation by helping the test designer, to choose 
items but do not print the test itself. When this approach is used, the questions need 
not be banked in computer retrievable form only the relevant item characteristics 
need to be accessible the computer. The computer provides the list of items identi¬ 
fiers. Tne items are themselves retrieved from manually assessed files and the 
test is formatted and printed. 

Computer Program for storage and Retrieval Process 

The main program 'STRT* stores all the questions found in the Question Bank 
with the 'ie+r.lied description of each question l. e. for each question it stores the vector 
A, The array of 7 components which are 

A (1) : The chapter in which the question lies 

A (2) Serial Number of the question as found in Bank 

A (3) Code of subject content within the chapter. 

A (4) Type of question 

A (5) Objective of question 

A (6) Time allocated to the Question (in Mts.) 

A (7) Marks allotted to the question. 

While storing the question on tape, it also calculates the components of vector 
N, whose ith component denotes the number of questions found in the ith chapter of the 
Bank, by comparing the value of A(l), for each question with the value of TN, the varia¬ 
ble according the previous question’s chapter code, and if they happen to be the same, 
the component of vector N is incremented, otherwise It shifts to the next component of 
vector N. 

Hie input cards, whose number is equal to the number of questions wanted to 
be taken, from the bank, are read one by one. Each card contains the desired discre¬ 
tion of question wanted to be taken out, which is stored in vector B of 6 components: 

B (1) : The chapter code from which the question is to be taken 

B (2) : Subject-content within that chapter 

B (3) : Type of Question Desired 
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B (4) : Objective of Question wanted to be taken out 

B (5) : Time given to this question 

B (6) Marks allotted to this question 


C MAIN PROGRAM STRT 

DIMENSION A(7), N(10),B(6), C(5000, 7) 

DATA NS, NT, NQ.NQD / 10, 580, 500, 15/ 

C STORAGE AND FINDING VECTOR N 

1= 1 

TN = NT 

DO 2 J = 1, NQ 

READ (3.20) (A (I), I = 1, 7) 

20 FORMAT (2 F 4. 0, F 5.1, 2 A 2, 2 F 3. 0) 

WRITE (3) A 

IF (TN. NE. A(l) ) GO TO 13 

5 N (I) = N(I) + 1 
GO TO 2 

13 1 = 1+1 
IN = A(l) 

GO TO 5 

2 CONTINUE 
PRINT 500 

50C FORMAT ( // 5 x, 'NUMBER OF QUESTIONS IN EACH CHAPTER //) 
PRINT 501, ( ( I, N (I), 1 = 1, NS) 

501 FORMAT (5 x, 2 I 8) 

C RETRIEVING AND FINDING QUESTIONS ACCORDING TO SET NORMS 

PRINT 14 

14 FORMAT ( // 5x, 'DESCRIPTION OF THE SET QUESTIONS’, 5x, 
TOTAL NUMBER OF QUESTION FOUND IN BANK OF THIS KIND') 

DO 5 I = 1, NQD 

REWIND (3) 

READ 21, B 

21 FORMAT (F 4. 0, F5.1,4A 2, 2F 3. 0) 

C FINDING TO WHICH CHAPTER THE QUESTION BELONGS 

LPP = B(i) 

LAB = LPP - NT - 1 
IF (IAB . EQ. 1) GO TO 6 
LA = IAB - 1 

6 DO 7 I = 1, LA 

7 IS = IS + N (I) 

C THIS IS A DUMMY READ STATEMENT TO REACH TO THAT POINT 

OF THE TAPE ON WHICH THE QUESTIONS OF THIS CHAPTER ARE 
STORED 
DO 8 J = 1, IS 
READ (3) A 

8 CONTINUE 
IS = IS + 1 
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GO TO 9 
IS = 1 
9 IZ = 0 

IP = N (LAB) 

DO 10 K = IS, IP 
READ (3) A 
DO 11 IK = 3, 7 
IL = IK - 1 

IF ( A (IK) .NE. B (IL) ) GO TO 10 
11 CONTINUE 
IZ = IZ + 1 
DO 12 IK = 1, 7 
12. C(IZ , IK) = A (IK) 

10 CONTINUE 

IF (IZ. EQ. 0) GO TO 200 
IF (IZ. EQ. 1) GO TO 201 
K = IZ / 2 

203 PRINT 13, (C (K, I), 1 = 1, 7), IZ 
13 FORMAT (2F 4. 0, F5.1, 2A2, 2F3. 0, 14) 

GO TO 5 

200 PRINT 202 

202 FORMAT (// 5x, ’NO QUESTION OF THESE SPECIFIED NORMS 
IS FOUND IN THE BANK’) 

GO TO 5 

201 K = 1 

GO TO 203 
5 CONTINUE 
STOP 
END 


Description of Variables Used 


Array of 7 components, giving the description of a question 

The vector of components, equal to the number of chapters 
in the Bank, whose ith component tells the number of questions 
in the ith chapter of Bank. 

INPUT vector of 6 components giving the description according 
to which the question has to be retrieved. 

Number of Chapters in the Bank 
Code of the 1st Chapter in the Bank 
Number of question in the Bank 
Number of questions wanted to be taken out 


With this kind of information, for each question computer firstly calculates the 
serial number of the chapter to which the question belongs and then calculating the number 
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of questions* which are lying before this specified chapter. The tape is moved forward 
by means of dummy read* so that the Retrieval - Process is carried on from the suita¬ 
ble point. 

After the tape is brought to the point where that chapter’s questions are stored, 
scanning is done for each question taken from that point uptill the end of that chapter. 
The questions found to have the norms found in the Bank of this specified description i.s 
noted and given as an output. Then a random selection is made from the obtained ques¬ 
tions and their identifying characteristic together with serial number as found in the bank 
is given as an output. 

Whether items are retrieved or generated, there are several possible versions. 

In many cases* exams, are printed directly on reproduction masters. In some systems 
the test can be printed in several scrambled versions (i. e. the same list of items). 

Benefits 


System which print tests have the obvious advantage of reducing the clerical 
chores of users and providing error free text. System which also aid in item selection 
make questions conveniently available, without the need for users to tediously search 
from the item text on item - characteristics. The educator can produce tests carefully 
tailored to meet the instructional needs as he defines. 

When the large number of questions become available then the problem of test 
security practically disappears.? The question - paper leakage scandals will end. These 
scandals arise from the secrecy that is at present sought to be maintained - often in 
vain. The question papers before and after printing pass through so many hands that 
secrecy Is very difficult to maintain. 

Thirdly, the system will be economical. We are satisfied that the system if 
used, surpass our minimum expectation for it - that of providing multiple versions 
of examination on duplicating masters to be typed, by a secretary and far less than 
the same amount of instructional material to be covered by on his inter active CA I, 
a typical test containing 100 items selected questions only throughout a bank required 
1. 07 minutes total computer time at a cost of $ 3.46, obviously a prudent use and this 
remains less expensive than its manual counter part. 

Surely this service with added feature of automatic scoring and item recorded 
maintenance will be seen by faculty as leading to further freedom for educational inno¬ 
vation, particularly in the areas, of the modern approach to self paced instruction in 
self testing and even in traditional lecture examination teaching. 


How far Accepted 

It might be added that the several universities in India have already introduced 
the ’Question Bank scheme. Bombay University one of the oldest in the country has 
initially decided that to begin with 15% of the questions will be selected from the Banks. 
Berhampur University has already introduced the scheme fully. Calicut University has 
also started the experiment though only for few courses. Patna University has establish- 
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ed ’question Bank for Physics initially. Rajasthan University has arranged a series of 
workshops for setting up these Banks. Madras, Mysore, Madurai, Jabalpur, Gujarat, 
South Gujarat, Poona, have all with the help of a series of Question Bank workshops 
conducted by AIU, developed their own Question Bank systems. Many other universi¬ 
ties are adopting the Question Banks develped by AIU. It is true that the experiment 
has just been started that too with only one aspect. None of the Indian Universities has 
yet started the storage and retrieval process with computer and for test preparation. 

Not only in India, the developing countries which are leading in these activities 
and where although more than one hundred educational institutions have developed, 
collection of question stored on the magnetic tape. It appears that the teachers at 
these institutions make only marginal use of them, partly because of old habits, 
partly because of publicity about the bank and pretty because of the mistaken belief 
that the operating knowledge of sophisticated gadgety is necessary. 

We are not discouraged by the slow acceptance of the usefulness °f item 
banks, if for no reason than that they have great promise in allowing us to do 
what we are already doing but for more students at various levels of progress, 
cheaper, more quickly and with fewer errors. 

It is well known that innovative products and methods require years even 
decades to achieve full acceptance. Macluchan’s observation that ’In the age of 
Xerox, every man is a publisher" is even more appropriate for information 
retrieval from computers. We think that awakening has occured but beast is still 
groggy. 
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RANDOM MATRICES GENERATION 


The fact that multiple-choice test items can be marked either one or zero depending 
on whether the examinee has opted for the key response or not, has lead the researcher a 
new point. 

Suppose a multiple choice test consisting of three items were taken by the four 
examinees where each of the item has three options (The result is immaterial for the 
number of options taken) Their options when compared with the key responses will yield 
the score matrix of order (4 x 3) it is found to be: 


Table of Scores 


Exam inees/item s 


2 

3 

4 


I II III 

0 0 1 

0 1 1 

0 0 0 

0 0 1 


(A) 


Thus an examinee scores on these three items will coincide with one of the 8(=2 3 ) 
possibilities. 


0 0 0 
0 0 1 
0 1 0 
0 1 1 
10 0 
1 0 1 
1 1 0 
111 


(B) 


If there are three items and four examinees. Then the matrix as shown in (A) will be 
(4x3). The fact that any row of this matrix is one of the rows as shown in (B) proves that 
there are: 


8 x 8 x 8 x 8 =4x8 

C 1 C 1 C 1 C 1 

possible ways of obtaining the score matrix as in (A): Then for each of the hypothetical 
configuration of the matrix (A) can be analysed for the objective type test analysis. 

The following computer program generates some of the hypothetical configuration 
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and finds the Mean, variance, together with the values of Kuder Richardson Reliability 
and Index of measurement efficiency. 

DESCRIPTION AND METHODOLOGY USED IN COMPUTER-PROGRAMME 

N : The number of Examinees 

M : The number of items 

IB : Matrix of order (2 M , M) 

PART I of the Program computes the matrix IB. 

Which is same as (B), shown in illustration, 

Where M = 3. 

IP1, IP2, IP3 : Vectors of length 2 used to generate the permutation 

of the elements 
1,2, 3,_, 2 W 

PART II of the Program generates the permutation with the help of SSP 
subroutine PPRCN, description of variables are as follows 

PPRCN (IP1, IP2, IP3, IK, 2, IER) 

where IP1,IP2, are the given permutation of integers from 1 to IK. 

IP3 is the generated permutation 

IK : length of any of the vectors IP1, IP2, or IP3 

M 

With the help of the generated permutation, IP3 of length 2 . The program splits 
the vector IP3 into KL (=2 M/n) parts. The First part NP1 which consists of the elements 


IP3 (I), IP3(2),-, IP3 (N) the second part NP2 which consists of the elements 

IP3(N+1), IP3(N+2),-, IP3(2N) and so on. 


Taking any part, say NPI, of IP3 computer generates a matrix AID of order 
(N x M) whose K*“ row is some as that of (NPT(K)p row of matrix IB. 

Thus with each permutation generated the program generates KL (=2 M /N) Matrices AID 
of order (N x M), Then for each. Matrix AID the statistical concepts MEAN, VARIANCE 
of Total scores is found with the help of the subroutine EMV, together with the values of 
Reliability as calculated by the Kuder - Richardson formula and that of Index of Meeure- 
ment Efficiency. 

PART HI computes the values of upper bound and lower bound (other than zero) 

i. Mean of scores 

ii. Variance of scores 

iii. K.R R. of scores 

iv. Standard deviation of scores 

v. Index of Measurement Efficiency. 

This work is accomplished by the subrontive AMA. 
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c 


MAIN PROGRAMME 


DIMENSION 

N-10 
M=5 
AM= M 
AN=N 

C PART I 

DO I J = 1, M 
IN = 2 * * (J - 1) 

K = 2 *■ * (M-J+l) 

JP = K - 1 
D04L = 1, JP,2 
K = (L - 1) * IN + 1 
IK = LfclN 
DO 11 I = K, DC 

11 IB(I, J) =0 
K = IK + 1 

IK = (L + 1)*IN 
Do 12 I = K, IK 

12 IB (I, J) = 1 

4 CONTINUE 

1 CONTINUE 

READ 41, (IP1 (I), I = l s IK), (IP2 (I), 1 = 1, IK) 

41 FORMAT ( ) 

PRINT 500 

500 FORMAT (IH2) 

C PART II 

L = K/N 

DO 110 KJ = 1,100 

CALL PPRCN (IP1, IP2, IP3, IK, 2, IER) 

DO 100 I = 1, L 
IZ = (I - 1) # N + 1 
IF = I * N 
K = 0 

DO 200 J = IZ, IF 
K = K + 1 
LJ = IP3 (J) 

DO 200 IP = 1,M 
200 AID (K, IP) = IB (U,IP) 

CALL EMV (AID, AM, AN, AEM, AV, AAK, ASD, AIME) 

EM (KJ,I) = AEM 

V (KJ, I) = AV 

AK (KJ, I) = AAK 

SD (KJ, I) = ASD 

BIME (KJ, I) = AIME 
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IOC CONTINUE 

DO 120 J = 1, DC 
IP2 (J) = IP3 (J) 

110 CONTINUE 

C PART in 

CALL AMA (EM. 100, L, BM, BSM) 

CALLAMA (V, 100, L, BV, SV) 

CALL AMA (AK, 100, L, BR, SR) 

CALL AMA (SD, 100, L, BSD, S SD) 

CALL AMA (BIME, 100, L, BIM, SIM) 

PRINT 7001 

7001 FORMAT (//20 x ’UPPER - LOWER SOUND AS CALCULATED FROM GENE¬ 
RATED MATRICES'//) 

PRINT 7002, BM, BSM, BV, SV, BR, SR, BSD, SSD, BIM, SIM, 

7002 FORMAT (5x, 'MEAN OF SCORE',6x,2F15.4,// 5x, 'VARIANCE OF 

1 SCORE',2x,2F15.4// 5x, 'K.R.R. OF SCORE', 6x,2F15.4,// 

2 5x, *S.D. OF SCORE',8x, 2F15.4, // 5x, 'I. M.E. OF SCORE', 

3 3x, 2F15.4, //) 

END 

SUBROUTINES USED 

SUBROUTINE EMV (AID, AM, AN, AEM, AV, AAK, ASD, AIME) 

DIMENSION AID ( ), S ( ) 

51 = 0. 

52 = 0. 

M = AM 
N = AN 

DO 1 I = 1, N 
S(I) = 0. 

DO 2 J = 1, M 

2 S(I) = S(D + AID (I,J) 

SI = SI + S(I) 

1 S 2 = S 2 +S(D*S(I) 

AEM = SI/AN 

AV = (S2 - AN * AEM * AEM) / AN 
Z1 = 0. 

DO 3 J = 1, M 

53 =0. 

DO 4 I = 1, N 
4 S3 = S3 + AID (I, J) 

3 Z1 = Z1 + S3 * (AN - S3) 

AAK = 1. - Z1 / (AN*AV*AN) 

AAK = AM*AAK / (AM - 1.) 
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ASD = SORT (AV) 

AIME = 1. - (.16#AM) /AV 
RETURN 

END 

SUBROUTINE AMA (EM, 100, L, BM, BSM) 
DIMENSION EM (100, L) 

BM= 0. 

BSM = 0. 

DO 1 I = 1, 100 
DO 1 J = 1, L 

IF (EM (I, J).EQ. 0.) GO TO 1 
IF (EM (I, J) . GT. BM) BM = EM (I, J) 
IF (EM (I, J) . LT. BSM) BSM = EM a. J) 

1 CONTINUE 
RETURN 
END 


OUTPUT OF THE PROGRAMME 

In our specific problem, computer generates 300 hypothetical configurations for 
the Multiple -choice test consisting of 5 items and 10 examinees. Programme analyses 
each of the generated score - matrix. 

Such results of 10 generated matrices is given below:- 


MEAN 

VARIANCE 

KUDER RICHARDSON 
RELIABILITY 

STANDARD 

DEVIATION 

INDEX OF MEA¬ 
SUREMENT EFFI¬ 
CIENCY 

2.5 

1.2500 

0. 0200 

1.1180 

0 3600 

2.2 

0. 7600 

-0.0625 

0.8718 

0 0526 

2.8 

1.1600 

0 . 0216 

1.0770 

0.3103 

1.7 

1 . 0100 

-0. 0990 

1.0050 

0.2079 

3.0 

1.4000 

0.2143 

1.1832 

0 4286 

2.9 

1.2900 

0.1357 

1.1358 

0 3798 

2.0 

0.4000 

-2.4375 

0.6325 

- 1.0000 

2.3 

1.6100 

0.3261 

1.2689 

0 5031 

1.9 

1.2900 

0.1744 

1.1358 

0 3798 

2.6 

1.2400 

0.1008 

1.1136 

0 3548 
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Here below, we present the table consisting of Upper/Lower bounds for the 
statistical concepts taken into analysis, (as based on 300 generated configurations) 


Upper/Lower Bound as Calculated from 
300 Random- Matrices 



Upper Bound 

Lower Bound 

Mean 

3.200 

0.0 

Variance 

2.2400 

0 . 0 

Kuder Richardson 
Reliability 

0. 5915 

-4.8957 

Standard -Deviation 

1.4967 

0 . 0 

Index of Measurement- 
Efficiency 

0.6429 

-2.3333 
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COMPUTER PROGRAMME FOR ANALYSIS OF OBJECTIVE TYPE TEST 


It is more useful to present a single general programme that will provide the 
teachers and researchers with a detailed and complete analysis of the test. The pro¬ 
gramme is written in Fortran IV on the IBM 360 and it is reasonably easy to adopt for 
use on any system having FORTRAN rv compiler. 

Data cards giving the following information has to be prepared. 

Data Card Information Stored 

No. 

1 Number of students (=N) and that of items (=M) 

2. Any set of N/2 integers from the integers 1 to N 

3 Values of Roll - Numbers of the N students 

4 KEY - RESPONSE OF M items 

5 A Matrix T of order (N X M) giving the N stu¬ 
dents options on the M items, (it has to be punched 
row wise) there will be N cards describing the 
matrix T. 

The working of programme is divided in six parts each part accomplishes the 
following task: 

PART FUNCTION 

1. Compares each student’s option on every item. With the key Response 

of the item and generates a matrix CM, whose (Uj)^ 1 element is 

= 1 ; if the ith student has opted for the Key Response on the 

jth item 

= 2 if it is left blank 

= 0 ; other wise 

it computes the total score of each student and splits the total 
according to First second half, odd - even split and Random half: 

2 Arranges the students in descending order w. r. t. the total score. 

Also changes Roll No, students’ option, and the matrix CM appropriately. 

3. Provides the Mark - Frequency table 
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4 Calculates some statistical concepts like Mean, Mode, Variance, 

S. D„ S. D. (By Paul Dieteich Method) and Median. 

5 Finds out value of Reliability of the test by various methods such 
as split halve (Spearman Brown), Rulon; Flangan, KR - 20, KR21, 
Tuckers' modified, Stanley Approximation and by Analysis of 
variance approach. 

6 Prepares the Derived score Table and the Question - analysis table. 
Question - Analysis Table consists of Choice, Mean Ability, Facility, 
Di scrimination indices. Derived Score table has Z, T, AGCT and 
CEEB scores of each examinee. 


C MAIN PROGRAMME 

INTEGERRNO 

REALOT, KR20, KR27, KR21 
DIMENSION 

DATA K/’H 1 / 

READ 411, N, M 

411 FORMAT ( ) 

K = N/2 

READ 412, (IRAN (I), 1 = 1, K) 

412 FORMAT ( ) 

AN = N 

AM = M 

READ 1, (RNO (I), 1=1, N) 

1. FORMAT (2014) 

READ2, (Q(I), 1 = 1, N) 

2 FORMAT (50A1) 

DO 311 = 1, N 

31 READ2, (T(I, J), J = 1, M) 

PRINT 500 

500 FORMAT (IH2) 

PRINT 60, (Q(J), J=l, M) 

60 FORMAT (2X, ’ORIGINAL DATA', //, 2X, 'KEY RESPONSE’, 2X, 50A1,//, 2X, 
1 'S. NO ', 5X, 'ROLL NO’, 5X, 'STUDENTS RESPONSE',///) 

DO 621 = 1, N 

62 PRINT 61,1, RNO (I), (T(I, J), J = 1, M) 

61 FORMAT (2X, 15, 5X, 15, 5X, 50(Ai, DC), 5X, F4. 0) 

C PART I 

DO 71= 1,N 
D07J = 1, M 

IF (T(I, J). EQ. Q(J))GOTO8 
IF (T(I,J).EQ.H)GOT09 
CM&J) = 0. 

GOT07 

8 CM(I, J) = 1. 
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G0T07 

9 CM(I, J) = 2. 

7 CONTINUE 

DOIOI = 1, N 
DOIOJ = 1, M 

IF(CM(I,J).EQ.2.)GOTO10 
RT(I) = RT(I) + CM(I,J) 

10 CONTINUE 
D011I = 1, N 
D012J - 1, M, 2 

IF (CM(I,J). EQ. 2. )GOTO!2 
OTa) = OT (I) + CM (I, J) 

12 CONTINUE 

11 ET(I) = RT(I)-OT(I) 

L = M/2 

D013I = 1, N 
D014J = 1,L 

IF (CM(I,J). EQ. 2.) GOTO!4 
FH (I) = FH (I) + CM (I, J) 

14 CONTINUE 

13 ALH(I) - RT(I)-FH(I) 

D015I = 1,N 
D016J = 1,L 

K - IRAN (J) 

IF(CM(I> K). EQ. 2.) G0T016 
RH(I) =■- RH (I) + CM (I» K) 

16 CONTINUE 

15 RHO (I) = RT(I) - RH (I) 

PRINT 500 

PRINT 50122 

50122 FORMAT (2X, 'S.NO.', 2X, 'ROLL NO', 5X, 'TOTAL',5X, 'ODD - EVEN', 
1 8X, 'FIRST-SECOND', 5X, 'RANDOM-HALF'//) 

DO50121I = l.N 

50121 PRINT5011,1, RNO(I), RT(I), OT(I), ET(I), FH(I), ALH(I), RH(I)t RHO(I) 

5011 FORMAT (2X,2I5, F5. 0,3(2(2X, F4. 0),3X)) 

CALL COR (FH, N, ALH, AK2, VF, V AL, VD2) 

CALL COR (RT, N, OT, AK4, VT, VO, VD4) 

CALL COR (OT, N, ET, AK1, VO, VE, VD1) 

CALL COR (RH, N, RHO, AK3, VRH, VRO, VD3) 

PART II 


JM = 1 
JIM = 1 
SMA = RT (I) 

BIG - RT(1) 

D08991I = 2, N 

IF (RT(I). GT. BIG)G0T08992 

IF (RT(I). LT. SMA)G0T08993 



GOTO8991 

8992 BIG = RT(I) 

JM = I 
G0T08991 

8993 SMA=RT(I) 

JIM-I 

8991 CONTINUE 

sam=sma' 

MRR=RNO(JM) 

MERR=RNO (JIM) 

D08900J = 1, M 
Z(J)=T(JM,J) 

CFMP(J) =CM(JM» J) 

TT(J) = CM(JIM, J) 

8900 AGCT(J) = T (JIM, J) 

D017I = 1, N 
IC = RT (I) 

JJ = 1 
IN = I + 1 
IRNO = RNO U) 

IRNO^RNO U) 

DO140J = 1, M 
CPMP(J) =CM(I, J) 

140 Tl(J) = Ta,J) 

D018J = IN, N 

IF(RT(J). GT. RT(I))G0T019 

G0T018 

19 RT(I) = RT (J) 

JJ = J 

18 CONTINUE 

IF(JJ.EQ.I)GOTO 17 

RT(JJ) - IC 

RNO (I) = RNO (JJ) 

RNO (JJ) = IRNO 
D0141J = 1, M 
CM(I, J) = CM(JJ, J) 

CM(JJ, J) = CPMP (J) 

T(I, J) = T (JJ,J) 

141 T(JJ,J) = T1 (J) 

17 CONTINUE 

RNO(l) = MRR 
RNO (N) = MIRR 
RT(1) = BIG 
RT(N) = SMA 
DO 6199J = 1, M 
T(l, J) = Z(J) 

T(N, J) = AGCT (J) 

CM(1, J) = CFMP (J) 
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6199 


CM(N, J) = TT(J) 

PRINT 500 
C PART III 

PRINT601 

601 FORMAT (5X, 'S.NO.5X, ’MARKS’, 5X,’FREQ 1 , 5X,’CUM FREQ',/////) 
PRINT600 
I = 0 

DO105J = 1, M 
F (J) = 0. 

AJ = J 

D0116IJ = 1, N 

IF(RT(IJ). EQ. AJ)F(J) - F(J) + 1. 

116 CONTINUE 

IF(F(J). EQ. 0. )GOTO105 
D023356KPL = 1 , J 
23356 CF(J) = CF (J) + F (KPL) 

1 = 1 + 1 

PRINT108,1, AJ, F(J), CF(J) 

105 CONTINUE 

108 FORMAT (2X,I8,5(5X, F8,2), 19) 

PRINTS 00 
PRINT82020 

82020 FORMAT (////, 5X, ’ORIGINAL DATA ARRANGED IN DESCENDING ORDER', //) 
DO82 022I = 1, N 

82022 PRINT61,1, RNO(I), (T(I, J), J = 1, N),RT(I) 

C PART IV 

D031I = 1, N 
SI =SI + RT(I) 

31 S2 = S2 + RT (I) * RT (I) 

AMEAN = SI /AN 

BIG = 0. 

D032J = 1, M 

IF(F(J). GT. BIG)G0T032324 
G0T032 

32324 BIG = F (J) 

L = J 

32 CONTINUE 
AMODE = L 
L = N/2 

IL = 2 * L 
J = L + 1 

IF (IL. EQ. N) GOT033 
AMED = RT (J) 

G0T034 
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33 AMED = .5 * (RT(L) + RT (J) ) 

34 VAR = (S2 - SI * AMEAN) / AN 
SD = SQRT (VAR) 

ALL = (AN/2.) + .5 
IP = (AN/6.) + .5 
D036I = 1, IP 
36 W1 = WI + RT (I) 

IT = N - IP + 1 
D038I = IT, N 
38 W3 = W3 + RT (I) 

SDPD =(W 1 - W3)/ALL 
SEM = SD / SQRT (AN - 1) 

C TO PRINT OUTPUT 

PRINT 500 
PRINT 66 

600 FORMAT (5X,48 (2H—)) 

PRINT 600 

66 FORMAT (//, 15X 'TABLE OF SOME DESIRED STATISTICS', ///) 

PRINT67, AMEAN, AMED, AMODE, VAR, SD, SDPD, SEM 

67 FORMAT (5X, ’MEAN OF SCORE', 17X, F10.4, ///, 5X, 'MEDIAN OF 

1 SCORE', 15X, FI0.4, //5X,'MODE OF SCORES', 12X, FI0.4,///, 5X,'VARIANCE 

2 OF SCORE', 10X,F10.4, ///, 5X,'S. D.', 10X, FI 0.4, //, 5X, 'S. D BY DIE TIC H 

3 METHOD’, 2X,F10.4, ///, 5X,'STANDARD ERROR OF MEAN', 2X, F10.4) 
PRINT600 

C PART V 

C CALCULATING RELIABILITIES BY VARIOUS METHODS 

SW1 = (2. * AK1) / (1. + AK1) 

SW2 = (2. * AK2) / (1. + AK2) 

SW3 = (2. * AK3) / (i. +AK3) 

ROE = 1. - VD1/VT 
RFS = 1. - VD2/VT 
FOE = 2. * (l.-(VO+VE)/VT) 

FFS = 2. *(1.-(VF + VAL)/VT) 

FRO = 2. * (1. -(VRH + VRO)/VT) 

RRO = 1. = VD3/VT 

C ANALYSIS OF VARIANCE APPROACH 

AF = (SI* SI)/(AM AN) 

SSE - S2/AM-AF 
D074J = 1, M 
D073I = 1,N 

IF(CM(I, J), EQ. 1. )GOTO75 
IF(CM(I, J). EQ. 0. )G0T076 
GOT073 

75 CR(J) = CR (J) + 1. 

GOTO73 

76 WR(J) = WR (J) + 1. 
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73 CONTINUE 

SC = SC + CR (J) 

SW = SW + WR (J) 

WAR (J) = WR (J) * CR (J)/(/N*AN) 

Z12 = Z12 + WR (J)* CR (J)/ (AN* AN) 

74 Z1 = CR(J) * CR (J) + Z1 
SSI = Zl/AN - AF 

TSS = (SC*SW)/(SC+SW) 

SSR = TSS - SSI - SSE 
Z1 = AM - 1. 

Z2 = AN - 1 

Z3 = Z1 * Z2 

AMSI = SSI /Z1 

AMSE = SSE / Z2 

AMSR = SSR / Z3 

VI = AMSI / AMSR 

VE m AMSE / AMSR 

RAV = (AMSE - AMSR) / AMSE 

PRINT 500 

PRINT 750 

750 FORMAT (20X, 'ANALYSIS OF VARIANCE TABLE', //) 

PRINT 600 
PRINT 760 

760 FORMAT ^X, 'SOURCE OF’, 5X, 'SUM OF SQUARES', 5X,’DEGRRES OF', 

1 5X, 'MEAN SQUARES', 5X,'VARIANCE RATIO’,/, 5X,'VARIATION' 5X, 

2 ’FREEDOM', 28X, 'ERROR') 

PRINT 600 

PRINT77, SSE, Z2, AMSE, VE, SSI, Z1, AMSI, VI, SSR, Z3, AMSR 
77 FORMAT (5X 'EXAMINEES', 5X, F15.4,//5X,'ITEMS', 8X, F15.4, 5X, F8. 0, 

1 10X, FI5.4, 5X, F15.4, //, 5X,'REMAINDER', 8X, F12.4, 5X, F8. 0,10X, FI5.4, 

2 5X, F15. 4, 2X) 

PRINT 500 
PRINT 68 

68 FORMAT(///,lOX, 'RELIABILITY OF THE TEST',////, 5X, 1. SPLIT HALVE', 

1 //) 

PRINT69, AK1, AK2, AK3 

69 FORMAT (8X 'ODD-EVEN SPLIT' 8X, F8.4, /, 8X, ’RANDOM HALF SPLIT’., 

1 5X, F8.4, /, 8X, 'FIRST-SECOND HALF', 3X,F8. 4,//) 

PRINT 70 

70 FORMAT (5X, '2. SPEARMAN BROWN WHOLE TEST RELIABILITY’, //) 
PRINT 69, SW1, SW2, SW3 

PRINT71 

71 FORMAT(5X, '3. RULON FORMULA OF RELIABILITY', //) 

PRINT 69, ROE, RFS, RRO 

PRINT72 

72 FORMAT (5X, ’4. FLANGANS FORMULA OF RELIABILITY’, //) 

PRINT600 

AMR = (AK4^ 3QRT(VT)-SQRT(VO))/SQRT(VT+VO-2> AK4 * SQRT(VT *VO)) 
PRINT730, AMR 
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730 FORMAT (///, 5X,'RE LIABILITY BY MOISER SHORT CUT METHOD’, F8.4) 

C INTERNAL CONSISTENCY RELIABILITY ESTIMATES 

S5 = 0. 

54 - 0. 

LI - AN * 27. /100. 

AAL = LI 
DO80J = 1, M 
WR27 (J) = 0. 

CR27 (J) = 0. 

D081I = 1, LI 

IF (CM (I, J). EQ. 1. )GOTO82 
IF(CM(I, J). EQ. 2. )GOT0811 
WR27(J) - WR27(J)+1. 

GOTOSll 

82 CR27(J) = CR27(J)+1. 

811 IF(J.EQ.l) GOTO8U12 

GOT081 

81112 S4 = S4 4 R T (I) * RT(I) 

55 = S5 + RT (I) 

81 CONTINUE 

IZZ = N - LI +1 

D0831 = IZZ,N 

IF(C M(I, J). EQ. 1. )G0T084 

IF (CM(I, J). EQ. 2. )G0T089 

ACR27(J) - AWR27(J) + 1. 

G0T089 

84 ACR27(J) = ACR27(J) + 1. 

89 IF(J. EQ. l)GOTO89909 

G0T083 

89909 S4 s S4 + RT(I) *RT (I) 

S5 - S5 + RT(I) 

83 CONTINUE 

80 SIGMA = SIGMA + (WR27(J)+AWR27(J))* (CR27(J) + ACR27(J))/(AAL . AAL *4 
AZ =2. * AAL 
S5 = S5 / AZ 

VAR27 = (S4 - AZ S5)/AZ 
KR20=AM HI• “Z12/VT)/(AN - 1.) 

KR27 = AM Ml - -SIGMA/VAR27)/(A M - 1.) 

AKR20 = AM * (AM *VT-AMEAN* (AM-AMEAN))/((AM-1.) * AM *VT) 

58 = 0. 

55 = 0. 

59 = 0. 

56 = 0. 

57 - 0. 

D085J = 1, M 

S6 = S6 -r WR(J)/AN 

S5 - S5 + CR(J)/AN 
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S8 = S8 +(CR (J) * WR(J)) /(AN * AN) 

85 S9 = S9 + (CR(J) * CR(J))/(AN* AN) 

XP = S5 / AM 
XQ = S6 / AM 

VARP = (S9 - AM * XP * XP)/AM 

TMKR = AM * (VT-AM *XP *XQ+AM *VARP)/ (VT* (AM-1.)) 

KR21 = AM (VT-AM* XP * XQ)/((AM-1.) VT) 

COEF = AM v (VT - S8) /((AM-1.) fc VT) 

C PART VI 

C QUESTION ANALYSIS TABLE 

SW = 0. 

D095J = 1,M 
D096I = 1,N 

IF(CM(I,J).EQ.2.)G0T097 
C(J) = C(J) + 1. 

97 IF(CM(I, J). EQ. 1. )AI(J) = AI(J) + RT(I) 

96 CONTINUE 

CI(J) = C(J) *100., AN 
A1(J) = AI(J)/C(J) 

FI(J) = (CR27(J) + ACE?7(J))/(2. * AAL) 

DI(J) = (CR27 (J) - ACR27(J))/AAL 
SW = SW + DI (J) * SQRT (VVAR(J)) 

SUD = SUD + DI (J) 

SFF = 3FF + FI (J)/100. 

SFF2 = SFF2 + FI(J) *FI (J)/lOOOO. 

95 CONTINUE 

ACOEFF = AM * (1.-S8/(SW * SW))/(AM-1.) 

SATR = AM* (1.-6. * (SFF - SFF2)/SUD)/(AM-1.) 

AIME = l.-(. 16 AM)/VT 

PR INTI 00, KR20, KR27, AKR20, KR21, COEF, COEFF, SATR, AIME 

100 FORMAT (///, 5x,'RELIABILITY BY KR-20 FORMULA’, 12x, F10.4 
1,/, 5x, ’KR-20 FORM. BASED ON 27% HAG AND 27% LAC?, F10.4 

2 , /5x, 'ANOTHER FORM OF KR-20 FORM', 15x, FI0.4, /, 5x, 

3 'RELIABILITY BY KR-21 FORM. ", 18x, FI 0.4, /, 5x, *CRONBACK 

4 COEF ALPHA’, 2Ox, FI0.4/, 5x, 'LOWER BOUND ESTIMATE OF 

5EXAM RELIABIT ITYMOx, F10.4/,5x, ’STANLEY APPROXIMATION 1 , 

6 22x, F 10.4, /, 5x ’INDEX OF MEASURE MENT EFFICIENCY 1 , F25.4///) 
PRINT66666, RAV, TMKR 

66666 FORMAT (5X, ’RELIABILITY BY ANALYSIS OF VAR. ’, 2X, F15.4, //, 5X, ’TUCKER 
MODIFIED K-R FORM.', 2X, FI5.4, ///) 

/ PRINT 600 

PRINT500 
PRINT 101 

101 FORMAT(//, 15X, ’QUESTION ANALYSIS TABLE’, ///) 

PRINT 600 

PRINT 102 

102 FORMAT(1X, ’QN. NO’,2X, ’%CHOOSING’, 2X, 'MEAN ABILITY’, 2X, ’FACILITY’ 
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1 , 2X, 'DISCRIMINATION', 2X, VARIANCE 1 , /IX, 'NO', 2X 'QUESTIONS 6X, 'INDEX , 

2 //) 

PRINT600 
DO103J = 1, M 

103 PRINT 104, J, Cl (J),AI(J),FI(J),DI(J),VVAR(J) 

104 FORMAT (I4,F12.4,2F 12.4,2X,2F 15.6) 

PRINT 600 

PRINT 500 
PRINT 109 

109 FORMAT (25X, 'DERIVED SCORE TABLE', ///) 

PRINT 600 
PRINT 6001 

6001 FORMAT (///,2X, »S.NO. ',2X, 'MARKS’, 2X, '2-SCORE', 2X, 'T-SCORE'. 

1 2X, *AGCT-SCORE' » 2X, 'CEEB - SCORE',/////) 

DO6099J = 1,N 

Z(J) = (RT(J) - AMEAN)/SD 

TT(J) - 50.+10. *Z(J) 

AGCT(J) = 2. * YT(J) 

CEEB(J) = 10. * TT(J) 

6099 PRINT6098, J,RT(J), Z(J), TT(J), AGCT(J), CEEB (J) 

6098 FORMAT (2X, 13,2X, F4. 0,4(2X, F8.2)) 

STOP 

END 

SUBROUTINE COR (X,N,Y,R, VX.VY.VD) 

DIMENSION X(I),Y(1) 

AN = N 
API = 0. 

AP2 = 0. 

apii= a 
api 2 = a 

D1 = 0. 

D2 = 0. 

DO 1 I = 1, N 
D = X (I) “ Yd) 

API = API + xa) 

AP2 = AP2 + Y a) 

AP11 = AP11 +X <I)*XT) 

AP22 = AP22 + Y (I) * Y (I) 

AP12 = AP12 + X (I)* Y(I) 

El =D1 +D 
D2 = D2 + D* D 

* VX = (AP11 - API * API/AN)/AN 
VY = (AP22 - AP2 * AP2/AN) /AN 
VD = (D2 - Dl* Dl/AN)/AN 

R = (AP12 - API * AP2/AN)/(AN* SORT (Vl^VY)) 

RETURN 

END 
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